Skip to content

Add zero-downtime-upgrade example - #175

Open
amiller wants to merge 3 commits into
Dstack-TEE:mainfrom
amiller:zero-downtime-upgrade
Open

amiller wants to merge 3 commits into
Dstack-TEE:mainfrom
amiller:zero-downtime-upgrade

Conversation

@amiller

@amiller amiller commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Adds zero-downtime-upgrade: upgrade a stateful app by starting a second CVM of the same app on the new compose and handing state to it, instead of rebooting the CVM in place.

  • upgrade.sh replicates the CVM with DORMANT=1 (forward-only), runs phala deploy --cvm-id on the replica alone with the new compose, waits for it to take the writer role, then deletes the old CVM.
  • The handover is two HMAC-authenticated calls (begin, commit); state is AES-GCM encrypted under a GetKey key, which both CVMs derive because they share the app id.
  • The role is persisted with the state so a restarted writer doesn't follow a deleted predecessor.

Measured on Phala Cloud (prod7, tdx.small) with a no-retry client: an in-place phala deploy --cvm-id failed 13 requests over ~65 s; two consecutive upgrade.sh runs failed 0 of 334, and a rerun of the simplified server 0 of 169. Forwarding through the retired CVM adds about 10 ms per request during the overlap.

Note for reviewers: the replica inherits allowed_envs, so the first deploy must name PREDECESSOR_URL and DORMANT (empty values are fine). Without that the replica boots as a second writer; upgrade.sh checks for it.

Also bump APP_VERSION with every code change: Compose doesn't recreate a container when only inline config content changes, so a code-only update keeps running the old code.

node zero-downtime-upgrade/test-local.mjs runs the protocol locally with a fake dstack socket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant